In meta-analyses, we can apply techniques to partially mitigate the risk of distortions caused by publication and reporting bias. Some of these strategies focus on improving the study search process, while others involve statistical adjustments.
Study Search - Several specific approaches can help minimize the risk of bias:
Include Grey Literature: such as dissertations, preprints, government reports, and conference proceedings;
Backward and Forward Snowballing: Review the references of included studies (backward snowballing) and identify papers that have cited those studies (forward snowballing).
Search in Multiple Databases
Comprehensive Search Strategies
Contacting Study Authors for Unpublished Data
Statistical methods: Statistical methods can help detect signs of publication bias by analyzing patterns in the data, such as asymmetry in funnel plots or a correlation between study size and effect size. While these methods cannot directly confirm publication bias, they highlight its possible presence. Some techniques also adjust for potential bias, providing corrected estimates of the overall effect size to improve the reliability of the meta-analysis
The reasoning is straightforward: Publication bias exists because SIGNIFICANT RESULTS are more likely to be published. Since LARGER studies have a HIGHER probability of producing significant results due to their larger sample sizes, publication bias disproportionately affects SMALL STUDIES, leading to an over-representation of large effect sizes in the published literature.
Studies with small sample sizes ➞ Low precision (high standard errors) ➞ Wider confidence intervals ➞ Greater likelihood of non-significant results.
This is precisely why methods for assessing publication bias focus on the issue of small-study effects. As the name suggests, these approaches are particularly concerned with the unique challenges posed by small studies.
It is common practice to assess small-study effects using funnel plots. A funnel plot is a scatter plot that displays:
Take a look at the example below:
In the absence of small-study effects (no publication bias), the studies should approximately follow the idealized funnel shape displayed in the plot. Is this true in our example? Subjectively, we can say that the studies do follow the shape reasonably well.
Studies located in the top part of the plot have low SE and should lie closely together, and not far away from the pooled effect size.
Studies located in the lower part of the plot, with increasing SE, the funnel “opens up”, and effect sizes are expected to scatter more heavily to the left and right of the pooled effect.
In this example there are 2 small studies with very low effect sizes to “balance out” the one with very high effects.
Now, let’s explore an example of funnel plot asymmetry:
In this case, the studies deviate from the idealized symmetrical funnel shape, suggesting the presence of small-study effects or potential publication bias.
There are three small studies with very high effect sizes located in the bottom-right corner of the plot (‘Shapiro’, ‘Kang’, and ‘Danitz-Orsillo’).
However, there are no corresponding small studies in the bottom-left corner of the plot with low or negative effect sizes to “balance out” these high-effect studies.
This imbalance contributes to the asymmetry in the funnel plot and may indicate the presence of small-study effects or publication bias.
Another concerning observation is that the study with the greatest precision in our sample, conducted by de ‘Vibe’, does not align well with the expected funnel pattern. Its effect size is noticeably smaller than anticipated for a study with such high precision.
Overall, the dataset exhibits an asymmetrical pattern in the funnel plot, which could suggest publication bias. It is possible that the three small studies with high effect sizes represent outliers that happened to achieve significant results, while a hidden pool of unpublished studies with similar standard errors but smaller, non-significant effects remains unaccounted for. This imbalance highlights the potential for selective reporting to skew the overall findings.
A good way to inspect how asymmetry patterns relate to statistical significance is to generate contour-enhanced funnel plots (Peterset al., 2008). Such plots can help to distinguish publication bias from other forms of asymmetry.
Usually, the statistical significance thresholds are 0.95 and 0.99, which equals p < 0.05 and p < 0.01, respectively. We see that the funnel plot now contains three shaded regions. We are particularly interested in the p ≥ 0.05, p < 0.05, and p < 0.01 regions.
In the funnel plot, the standard error of the treatment estimates is plotted on the y-axis by default which is likely to be the best choice (Example A). However, we also can plot on the y-axis the 1/SE (precision), as demonstrated in Example B.
Studies appear to be MISSING in areas of statistical NON-SIGNIFICANT or SIGNIFICANT NEGATIVE/LOWER effect (flattened in orange) ➤ Increase the possibility of that the asymmetry is due to publication bias.
See the example below:
Studies appear to be MISSING in areas of HIGHER STATISTICAL SIGNIFICANCE (flattened in pink) ➤ Reduces the plausibility that publication bias.
See the example below:
Look after other sources of ‘begin’ funnel plot asymmetry:
Between-study heterogeneity;
Study procedures were different in small studies;
High risk-of-bias - Low quality studies;
Random (by chance).
We will now make this more concrete by generating a funnel plot our
selves. In the meta package, the funnel()
function can be used to print a funnel plot for a meta-analysis object.
Here, we produce a funnel plot for our metabin()
meta-analysis object.
library(meta)
library(readxl)
library(dmetar)
## Author Year event.e n.e event.c n.c p2y12_type
## 1 Almendro-Delia et al. 2015 188 3973 116 2076 Clopidogrel
## 2 ATLANTIC Trial 2014 30 906 19 952 Ticagrelor
## 3 Danchin et al. 2018 9 360 18 360 Clopidogrel
## 4 Load&Go Trial 2013 2 56 0 56 Clopidogrel
## 5 CIPAMI Trial 2012 1 164 4 171 Clopidogrel
## 6 Fabris et al. 2022 17 237 17 300 Clopidogrel
## 7 Postma et al. 2014 46 1345 35 1096 Clopidogrel
## 8 Dorler et al. 2011 55 1635 110 1076 Clopidogrel
## 9 Lupi et al. 2016 5 143 13 143 Ticagrelor
## 10 Redfors et al. 2019 1960 37840 528 6964 Clopidogrel
## 11 Yudi et al. 2018 25 892 96 1915 Clopidogrel
## 12 Alexander et al. 2008 12 1029 55 2756 Ticagrelor
## 13 Fefer et al. 2009 12 217 6 166 Clopidogrel
## 14 Lev et al. 2008 1 165 2 127 Clopidogrel
## overal_risk_of_bias pretreatment _timing %_HTN
## 1 Serious-risk of Bias Pre-hospital 30.0
## 2 Low-risk of Bias Pre-hospital 28.8
## 3 Low-risk of Bias Pre-hospital 25.9
## 4 Moderate-risk of Bias Pre-hospital 52.3
## 5 Low-risk of Bias Pre-hospital 49.8
## 6 Moderate-risk of Bias Pre-hospital 30.5
## 7 Serious-risk of Bias Pre-hospital 20.3
## 8 Moderate-risk of Bias Pre-hospital 40.1
## 9 Serious-risk of Bias Pre-hospital 60.6
## 10 Moderate-risk of Bias Pre-hospital 40.5
## 11 Moderate-risk of Bias Pre-hospital 43.4
## 12 Serious-risk of Bias In-hospital 61.4
## 13 Serious-risk of Bias In-hospital 32.6
## 14 Serious-risk of Bias In-hospital 70.1
m.acm <- metabin(event.e, n.e, event.c, n.c,
data = ma$acm,
method = "MH",
method.tau = "DL",
sm = "RR",
studlab = Author)
summary(m.acm)
## RR 95%-CI %W(common) %W(random)
## Almendro-Delia et al. 0.8469 [0.6761; 1.0608] 11.0 13.0
## ATLANTIC Trial 1.6591 [0.9407; 2.9262] 1.3 8.3
## Danchin et al. 0.5000 [0.2277; 1.0981] 1.3 5.9
## Load&Go Trial 5.0000 [0.2455; 101.8321] 0.0 0.6
## CIPAMI Trial 0.2607 [0.0294; 2.3078] 0.3 1.2
## Fabris et al. 1.2658 [0.6606; 2.4255] 1.1 7.3
## Postma et al. 1.0710 [0.6951; 1.6502] 2.8 10.1
## Dorler et al. 0.3291 [0.2403; 0.4506] 9.6 11.8
## Lupi et al. 0.3846 [0.1408; 1.0508] 0.9 4.3
## Redfors et al. 0.6832 [0.6227; 0.7495] 64.4 14.2
## Yudi et al. 0.5591 [0.3626; 0.8619] 4.4 10.1
## Alexander et al. 0.5844 [0.3142; 1.0867] 2.2 7.6
## Fefer et al. 1.5300 [0.5864; 3.9914] 0.5 4.6
## Lev et al. 0.3848 [0.0353; 4.1970] 0.2 1.0
##
## Number of studies: k = 14
## Number of observations: o = 67120 (o.e = 48962, o.c = 18158)
## Number of events: e = 3382
##
## RR 95%-CI z p-value
## Common effect model 0.6887 [0.6380; 0.7433] -9.57 < 0.0001
## Random effects model 0.7274 [0.5673; 0.9327] -2.51 0.0121
##
## Quantifying heterogeneity:
## tau^2 = 0.1108; tau = 0.3328; I^2 = 73.7% [55.4%; 84.5%]; H = 1.95 [1.50; 2.54]
##
## Test of heterogeneity:
## Q d.f. p-value
## 49.46 13 < 0.0001
##
## Details on meta-analytical method:
## - Mantel-Haenszel method (common effect model)
## - Inverse variance method (random effects model)
## - DerSimonian-Laird estimator for tau^2
## - Mantel-Haenszel estimator used in calculation of Q and tau^2 (like RevMan 5)
## - Continuity correction of 0.5 in studies with zero cell frequencies
funnel(m.acm,
studlab = TRUE,
bg = "lightblue",
cex = 2,
cex.studlab = 1,
random = T,
fixed = F,
backtransf = F)
title("Funnel Plot for All-Cause Mortality")
# Define fill colors for contour
funnel(m.acm,
studlab = TRUE,
bg = "lightblue",
cex = 2,
cex.studlab = 1,
random = T,
fixed = F,
backtransf = F,
contour = c(0.95, 0.99))
legend(x = 1.65, y = 0.01,
legend = c("p ≥ 0.05", "p < 0.05", "p < 0.01"),
fill = c("white" , "#808080", "#E6E6E6"))
title("Contour-Enhanced Funnel Plot for All-Cause Mortality")
Tests for funnel plot asymmetry evaluate whether there is a systematic relationship between intervention effects and study size beyond what would occur by chance, indicating potential small-study effects.
While visual inspection of contour-enhanced funnel plots can highlight “red flags” for publication bias, such as asymmetry, this approach is subjective and lacks explicit rules for defining “too asymmetric.”
Quantitative tests provide a more objective assessment, complementing visual interpretation.
Tests for funnel plot asymmetry should only be used when a meta-analysis includes at least 10 studies, as fewer studies lack sufficient power to distinguish random variation from true asymmetry.
These tests are also inappropriate when all studies are of similar sizes (i.e., similar standard errors), as this limits their ability to detect asymmetry.
When small-study effects are identified, publication bias should be considered one of several possible explanations. While funnel plots and asymmetry tests highlight potential issues, they do not offer solutions.
Review authors should note that these tests have low power, so a lack of evidence for asymmetry does not rule out the presence of bias, including publication bias.
| Scenarios | Methods | R Syntax |
|---|---|---|
| Mean Differences | Egger 1997 | "Egger" |
| Odds Ratios | Harbord 2006 or Peters 2006 | "Harbord" | "Peters" |
Risk Ratios Risk Differences |
Egger 1997 or Peters 2006 | "Egger" | "Peters" |
| Standardized Mean Differences | Pustejovsky & Rodgers 2019 | "Pustejovsky" |
| Meta-analysis of diagnostic test accuracy studies | Deeks 2005 | "Deeks" |
To apply the tests to detect asymmetry we will use the
metabias() function from the meta package. We
need to provide some instructions for the metabias()
function.
metabias(m.acm,
method.bias = "Egger",
plotit = T)
## Linear regression test of funnel plot asymmetry
##
## Test result: t = 0.34, df = 12, p-value = 0.7363
## Bias estimate: 0.2498 (SE = 0.7246)
##
## Details:
## - multiplicative residual heterogeneity variance (tau^2 = 4.0812)
## - predictor: standard error
## - weight: inverse variance
## - reference: Egger et al. (1997), BMJ
After running the metabias() function, the next step is
to annotate the contour-enhanced funnel plot with the results from the
test used to assess symmetry. Here’s how to proceed:
funnel(m.acm,
studlab = TRUE,
bg = "lightblue",
cex = 2,
cex.studlab = 1,
random = T,
fixed = F,
backtransf = F,
contour = c(0.95, 0.99))
title("Contour-Enhanced Funnel Plot for All-Cause Mortality")
legend(x = 1.65, y = 0.01,
legend = c("p ≥ 0.05", "p < 0.05", "p < 0.01"),
fill = c("white" , "#808080", "#E6E6E6"))
legend(x = -2.9, y = 1.39, bty = "n", cex = 0.8,
legend = c(expression(bold("p-val = 0.74 (Egger's Test)")), "intercept: 0.25 (SE = 0.7246) | t = 0.34"))
The intercept (bias estimate) represents the expected standardized effect size (z -score) when study precision is zero (i.e., infinite standard error).
A value close to zero suggests symmetry in the funnel plot, indicating no small-study effects or publication bias.
Since the intercept is not significantly different from zero, the funnel plot is likely symmetrical.
Symmetry in the funnel plot suggests that small-study effects or publication bias are unlikely to have distorted the results.
data(Fleiss1993bin)
m1 <- metabin(d.asp, n.asp, d.plac, n.plac, data = Fleiss1993bin, sm = "OR")
m1
## Number of studies: k = 7
## Number of observations: o = 28003 (o.e = 14186, o.c = 13817)
## Number of events: e = 4414
##
## OR 95%-CI z p-value
## Common effect model 0.8969 [0.8405; 0.9570] -3.29 0.0010
## Random effects model 0.8683 [0.7559; 0.9973] -2.00 0.0457
##
## Quantifying heterogeneity:
## tau^2 = 0.0147 [0.0000; 0.1145]; tau = 0.1214 [0.0000; 0.3384]
## I^2 = 39.7% [0.0%; 74.6%]; H = 1.29 [1.00; 1.99]
##
## Test of heterogeneity:
## Q d.f. p-value
## 9.95 6 0.1269
##
## Details on meta-analytical method:
## - Mantel-Haenszel method (common effect model)
## - Inverse variance method (random effects model)
## - Restricted maximum-likelihood estimator for tau^2
## - Q-Profile method for confidence interval of tau^2 and tau
forest(m1,
layout = "Revman5",
sortvar = TE,
lab.e = "Experimental", label.left = "Favors Experimental",
lab.c = "Control", label.right = "Favors Control",
ff.lr = "bold",
leftcols = c("studlab","event.e", "n.e", "event.c", "n.c", "w.random", "effect", "ci" ) ,
leftlabs = c("Studies", NA, NA, NA, NA, NA, NA, NA),
text.random = "Random effects model",
random = TRUE,
common = FALSE,
test.overall.random = TRUE,
rightcols = FALSE,
colgap = "3mm",
fs.heading = 12,
fs.study = 12,
fs.hetstat = 10,
digits = 2,
digits.pval = 2,
pooled.events = TRUE,
pooled.totals = TRUE,
print.I2.ci = TRUE,
col.square="darkcyan", col.square.lines="black",
prediction = TRUE, col.predict = "#CEF2EE",col.predict.lines = "black", ff.predict = 1)
tf1<-trimfill(m1)
summary(tf1)
## OR 95%-CI %W(random)
## 1 0.7197 [0.4890; 1.0593] 6.8
## 2 0.6808 [0.4574; 1.0132] 6.5
## 3 0.8029 [0.6065; 1.0629] 10.8
## 4 0.8007 [0.4863; 1.3186] 4.5
## 5 0.7981 [0.5526; 1.1529] 7.4
## 6 1.1327 [0.9347; 1.3728] 16.4
## 7 0.8950 [0.8294; 0.9657] 26.8
## Filled: 5 1.0466 [0.7246; 1.5118] 7.4
## Filled: 1 1.1607 [0.7886; 1.7084] 6.8
## Filled: 2 1.2271 [0.8244; 1.8263] 6.5
##
## Number of studies: k = 10 (with 3 added studies)
## Number of observations: o = 31987 (o.e = 16369, o.c = 15618)
## Number of events: e = 4775
##
## OR 95%-CI z p-value
## Random effects model 0.9228 [0.8228; 1.0350] -1.37 0.1699
##
## Quantifying heterogeneity:
## tau^2 = 0.0113 [0.0000; 0.1145]; tau = 0.1061 [0.0000; 0.3384]
## I^2 = 37.4% [0.0%; 70.1%]; H = 1.26 [1.00; 1.83]
##
## Test of heterogeneity:
## Q d.f. p-value
## 14.37 9 0.1099
##
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-Profile method for confidence interval of tau^2 and tau
## - Trim-and-fill method to adjust for funnel plot asymmetry (L-estimator)
funnel(tf1,
studlab = TRUE,
bg = "lightblue",
cex = 2,
cex.studlab = 1,
random = T,
fixed = F,
backtransf = F,
contour = c(0.95, 0.99))
forest(tf1,
layout = "Revman5",
sortvar = TE,
lab.e = "Experimental", label.left = "Favors Experimental",
lab.c = "Control", label.right = "Favors Control",
ff.lr = "bold",
leftcols = c("studlab","event.e", "n.e", "event.c", "n.c", "w.random", "effect", "ci" ) ,
leftlabs = c("Studies", NA, NA, NA, NA, NA, NA, NA),
text.random = "Random effects model",
random = TRUE,
common = FALSE,
test.overall.random = TRUE,
rightcols = FALSE,
colgap = "3mm",
fs.heading = 12,
fs.study = 12,
fs.hetstat = 10,
digits = 2,
digits.pval = 2,
pooled.events = TRUE,
pooled.totals = TRUE,
print.I2.ci = TRUE,
col.square="darkcyan", col.square.lines="black",
prediction = TRUE, col.predict = "#CEF2EE",col.predict.lines = "black", ff.predict = 1)
For regression test results, it is usually sufficient to report the value of the intercept, its standard error, as well as the 𝑡 and 𝑝 value. It is common practice to include them alongside visual representations, such as a funnel plot, to provide a clearer context for interpretation.
Examples:
“The presence of small-study effects was assessed by visual assessment of the contour-enhanced funnel plots and formal Egger regression test.1-3”
“We explored the potential for publication bias by visual inspection of the contour-enhanced funnel plot and Peters’ test.1-3”
Examples:
“There was no evidence of small-study effects by visual assessment of the contour-enhanced funnel plots or by Egger regression test (see Supplementary Figure X).”
“Contour-enhanced funnel plot analysis and Egger regression test detected evidence of publication bias for the primary endpoint (intercept = 1.25, SE = 0.52, t = 2.40 , p = 0.001 ), suggesting potential small-study effects or selective reporting (see Supplementary Figure X).”
“We did not assess publication bias due to the limited number of studies included (n < 10), as regression-based methods lack sufficient power to distinguish chance from true asymmetry.”
Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol. 2008;61(10):991-6.
Sterne JA, Sutton AJ, Ioannidis JP, Terrin N, Jones DR, Lau J, Carpenter J, Rücker G, Harbord RM, Schmid CH, Tetzlaff J, Deeks JJ, Peters J, Macaskill P, Schwarzer G, Duval S, Altman DG, Moher D, Higgins JP. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011 Jul 22;343:d4002.
Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997 Sep 13;315(7109):629-34.
Harbord RM, Egger M, Sterne JA. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med. 2006 Oct 30;25(20):3443-57.
Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Comparison of two methods to detect publication bias in meta-analysis. JAMA. 2006 Feb 8;295(6):676-80.
Pustejovsky JE, Rodgers MA. Testing for funnel plot asymmetry of standardized mean differences. Res Syn Meth. 2019; 10: 57–71.
Deeks JJ, Macaskill P, Irwig L. The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. J Clin Epidemiol. 2005 Sep;58(9):882-93.